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Introduction 


There  are  recurring  patterns  or  forms  that  are  found  among  expert  analyses. 
Some  of  the  different  forms  that  these  analyses  take  are  cost-benefit  analysis,  stage 
models,  hierarchies,  trends,  and  multi-factor  models.  Learning  to  analyze  complex 
situations  implies  mastering  how  to  carry  out  investigations  of  phenomena  guided  by 
one  or  more  of  these  target  structures.  We  refer  to  these  target  structures  that  guide 
analysis  as  epistemic  forms,  and  the  set  of  rules  and  strategies  that  guide  analysis  as 
epistemic  games. 

Everyone  engages  in  the  most  basic  epistemic  games:  list  making,  compare  and 
contrast,  constructing  causal  chains,  determining  the  steps  in  a  process,  analyzing 
trends,  etc.  Research  and  practice  have  elaborated  these  basic  strategies  by  adding 
specialized  constraints.  By  exploiting  these  constraints  systematically,  experts  can  more 
productively  explore  any  domain  of  inquiry. 

In  prior  work  we  have  constructed  a  preliminary  theory  of  the  epistemic  forms 
and  games  experts  use  to  conduct  inquiries  (Collins  &  Ferguson,  1993;  Morrison  & 
Collins,  1995).  In  our  current  work  for  ARI  we  studied  how  scientists  and  military 
analysts  make  sense  of  complex  situations.  Based  on  our  analysis  of  these  data,  we 
have  developed  an  elaborated  theory  of  epistemic  forms  and  games,  which  can  form  the 
basis  for  building  a  tool  to  support  expert  analyses. 

We  would  argue  that  this  research  will  be  useful  both  in  developing  expert 
advisory  systems  and  in  training  military  commanders.  For  example,  the  kind  of 
system  we  propose  to  develop  could  be  used  to  provide  both  tools  and  advice  to 
experts  for  analyzing  complex  situations.  With  respect  to  training,  we  would  argue  that 
it  would  be  useful  for  future  military  commanders  to  learn  some  of  the  most  important 
epistemic  forms  and  games  to  guide  their  analyses.  They  have  value  in  any  endeavor  to 
understand  the  world. 
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Prior  Work 


This  section  briefly  summarizes  the  preliminary  theory  developed  in  Collins  and 
Ferguson  (1993).  We  divide  epistemic  games  into  three  types:  structural-analysis 
games,  functional-analysis  games,  and  process-analysis  games. 

Structural-Analysis  Games 

One  of  the  simplest  epistemic  games  is  list-making.  People  often  make  lists  as 
part  of  carrying  out  their  day-to-day  activities,  but  they  also  make  lists  in  an  attempt  to 
understand  the  world.  Every  list  is  implicitly  the  answer  to  a  question.  Some  epistemic 
questions  might  be  "What  are  the  basic  substances  things  are  made  of?"  "What  are 
possible  courses  of  action  to  win  a  battle?,"  and  "What  were  the  causes  of  the  French 
Revolution?"  If  the  answer  to  these  questions  must  be  discovered,  rather  than  recalled 
or  looked  up,  then  the  list-making  process  is  an  inquiry  process  and  the  resulting  list 
constitutes  new  knowledge. 

List-making  can  be  elaborated  by  adding  constraints  on  the  contents  of  the  list. 
These  constraints  are  the  rules  of  the  game  and  serve  two  purposes.  They  cause  the 
resulting  list  to  be  more  focused  and  they  facilitate  the  finding  of  ideas.  The  constraints 
established  by  the  list-making  game  that  we  have  identified  are:  similarity,  coverage, 
distinctness,  multiplicity,  and  brevity.  Similarity  is  the  requirement  that  the  items  in 
the  list  be  of  the  same  general  form:  the  same  scale,  the  same  kind  of  thing,  of  the  same 
importance,  and  so  on.  Coverage  means  that  all  possible  answers  to  the  question  are 
covered  by  the  items  on  the  list.  Distinctness  requires  that  no  two  items  overlap  or  are 
difficult  to  distinguish.  Multiplicity  means  that  a  list  must  have  more  then  one  element. 
Brevity  refers  to  the  fact  that  short  lists  are  generally  better  than  long  ones,  because  they 
constitute  more  succinct  answers  to  the  inquiry. 

Each  of  these  constraints  leads  to  useful  list-constructing  strategies  in  the  form  of 
auxiliary  questions  that  may  help  to  guide  the  inquiry.  Similarity  provokes  the 
question:  "Is  one  of  these  things  not  like  the  others?"  Coverage  asks  "Has  anything  been 
left  out?"  or  "Is  every  example  I  can  think  of  covered  by  one  of  the  items  in  the  list  so 
far?"  Distinctness  leads  to  asking  "Do  any  of  these  items  overlap  or  mean  the  same 
thing?"  Multiplicity  is  a  definitional  constraint  and  really  only  leads  to  the  question 
"Am  I  really  seeking  a  list?"  when  only  one  item  can  be  thought  of.  Finally,  brevity 
(when  a  list  begins  to  grow  too  large)  prompts  questions  like:  "Should  I  be  using  more 
abstract  categories?"  or  "Can  the  elements  of  this  list  be  partitioned  in  some  way?"  The 
questions  generated  by  violations  of  the  brevity  constraint  often  lead  to  major  shifts  in 
the  nature  of  what  is  being  listed.  The  magnitude  of  the  developing  list  may  push  one 
into  deciding  to  use  much  larger  classes,  or  to  change  games  altogether  by  trying  to 
form  a  hierarchy  or  a  table. 
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The  desired  result  of  any  epistemic  game  is  the  completion  of  an  epistemic  form 
that  satisfies  the  inquiry.  Because  of  this  correspondence,  the  names  of  the  games  and 
forms  are  often  similar  —  the  list-making  game  produces  lists,  the  system-dynamics 
game  produces  system-dynamics  models,  and  compare  and  contrast  produces  a 
comparison  table.  But  the  same  form  may  be  produced  by  more  than  one  game;  for 
example  the  primitive-elements  game  also  produces  lists. 

The  list  game  is  the  simplest  structural-analysis  game,  but  there  are  others.  The 
most  similar  to  the  list  game  are  spatial  and  temporal  decompositions  (i.e.  stage 
models).  More  complicated  structural-analysis  games  include  compare-and-contrast, 
cost-benefit  analysis,  primitive-elements  analysis,  tables  or  cross-product  analysis,  and 
tree  structures  or  hierarchical  analysis.  Structural-analysis  games  answer  the  question 
"What  is  the  nature  of  X?"  by  breaking  X  down  into  subsets  or  constituents  and 
describing  the  relationships  among  the  constituents.  We  briefly  describe  each  of  these 
games. 

Spatial  decomposition  is  the  kind  of  analysis  that  takes  place  in  anatomy 
or  circuit  diagrams.  The  goal  is  to  break  an  entity  down  into  a  set  of  non¬ 
overlapping  parts,  and  to  specify  the  topographical  relations  between  the  parts. 

The  set  of  constraints  is  the  same  as  the  list-making  game,  though  each  has  a 
spatial  aspect.  Specifying  the  topographic  connections  is  an  additional 
constraint,  and  where  applicable,  specifying  the  nature  of  these  connections  is 
another  constraint. 

Temporal  decompositions  or  stage  models  are  common  in  historical  analysis, 
psychological  analysis,  and  analysis  of  any  process  that  is  characterized  by  a  series  of 
states.  The  simplest  stage  model  is  a  list  constructed  with  the  constraint  that  the  stages 
follow  each  other  sequentially  without  overlap. 

Figure  1  shows  a  more  complicated  version  of  a  stage  model,  and  illustrates  the 
way  we  represent  epistemic  forms.  Each  stage  might  be  characterized  by  multiple 
characteristics,  and  furthermore  these  characteristics  may  be  arranged  on  a  set  of 
dimensions  (e.g.,  the  boy  was  angry  and  tired  before  his  nap,  but  happy  and  energetic 
afterwards).  In  a  more  complicated  stage  model,  the  interrelationship  between  the 
variables  might  be  specified  (e.g.,  energy  state  determines  mood),  and  the  reason  for  the 
change  from  one  stage  to  the  next  specified  (e.g.  a  nap  leads  to  an  increase  in  energy 
state).  These  last  four  constraints  (i.e.,  multiple  characteristics,  specified  dimensions, 
specified  interrelationships,  and  reasons  for  transition)  are  all  optional  constraints  that  a 
person  might  or  might  not  use  in  constructing  a  stage  model. 
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Figure  1.  Stage  Models 

Compare  and  contrast  is  a  decomposition  game  involving  comparison  of  two 
entities.  It  is  commonly  used  in  beginning  analysis  in  many  different  fields,  but  is 
perhaps  most  prominently  seen  in  history  and  the  social  sciences.  The  most  common 
version  of  this  game  employs  dimensional  analysis,  where  the  common  and  distinctive 
features  are  identified  with  particular  dimensions. 

Cost-benefit  analysis  is  a  special  case  of  compare-and-contrast,  that  is  used  in 
social  and  economic  policy  analysis.  The  things  compared  in  cost-benefit  analysis  are 
alternative  courses  of  action,  and  in  playing  this  game  one  should  first  identify  all 
possible  courses  of  action  (i.e.  the  coverage  constraint  in  the  list  game).  Then  one  tries 
to  identify  all  the  costs  and  benefits  (or  pros  and  cons)  of  each  alternative.  This  might 
also  involve  a  set  of  dimensions  on  which  the  alternatives  are  compared,  such  as  time, 
effort,  and  money.  Identifying  all  the  costs  and  benefits  (i.e.  to  obtain  coverage)  for 
each  alternative  is  aided  by  knowing  about  likely  kinds  of  costs  and  benefits,  such  as 
time,  money,  and  effort;  it  is  also  aided  by  knowing  to  look  for  possible  side  effects,  for 
social  as  well  as  individual  effects,  and  for  possible  countereffects  and  synergies. 

The  primitive-elements  game  is  a  version  of  the  list  game  that  has  driven  much 
of  the  history  of  the  physical  sciences  and  now  is  playing  a  large  role  in  artificial 
intelligence  analyses  of  the  social  sciences  (e.g.  Schank  &  Abelson,  1977;  Waltz,  1975). 
Chemistry  identified  92  natural  elements,  and  when  atoms  were  discovered,  the  quest 
was  to  determine  the  basic  elements  from  which  atoms  are  made.  The  latest  attempt  is 
to  determine  different  types  of  the  primitive  elements  (i.e.  quarks)  making  up  protons, 
neutrons,  and  other  subatomic  particles.  Coverage  of  all  the  phenomena  by  the  set  of 
primitive  elements  is  particularly  critical  in  this  game.  Another  constraint  is  to  specify 
how  the  elements  combine  to  produce  each  phenomenon. 
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The  cross-product  or  table  game  is  a  multi-dimensional  version  of  the  list  game. 
The  best  known  example  of  the  cross-product  game  was  the  construction  by  Mendeleev 
of  the  periodic  table  of  chemical  elements.  This  led  to  identifying  missing  elements  and 
ultimately  to  an  understanding  of  the  atomic  structure  of  molecules.  The  dimensions 
can  be  continuous  or  discrete,  and  cells  can  multiply  filled  or  not.  These  latter 
constraints  have  a  large  effect  as  to  how  the  cross-product  game  is  played.  But  as  in  the 
primitive-elements  game,  coverage  of  all  the  elements  is  a  critical  constraint  in  playing 
the  cross-product  game. 

The  tree-structure  or  hierarchy  game  is  familiar  to  everyone  from  its  use  in 
biology.  Often  the  tree-structure  game  is  employed  as  a  transfer  from  the  list  game 
when  a  list  gets  too  long.  The  added  constraints  in  a  tree  structure  are  that  the  elements 
be  broken  into  subsets  of  similar  types  (the  similarity  constraint)  and  the  relations 
among  the  subsets  be  specified.  These  kinds  of  hierarchies  pervade  the  biological  and 
social  sciences,  since  evolutionary  processes  naturally  produce  tree  structures. 

These  are  the  most  common  structural-analysis  games.  Most  employ  the 
constraints  described  for  the  list  game,  but  each  adds  new  constraints.  They  are  the 
basis  for  understanding  the  structure  of  systems,  and  this  is  a  major  form  of  inquiry  that 
is  carried  out  in  analyzing  situations. 

Functional-Analysis  Games 

A  second  major  form  of  analysis  is  functional  analysis.  The  goal  in  this  kind  of 
analysis  is  to  determine  the  causal  or  functional  structures  that  relate  elements  in  a 
system.  The  most  common  functional-analysis  games  include  critical-event  analysis, 
cause-and-effect  analysis,  problem-centered  analysis,  multi-factor  analysis,  and  form- 
and-function  analysis. 

Critical-event  analysis  occurs  in  historical  analysis  and  troubleshooting  of 
various  kinds  (often  called  critical-incident  analysis).  This  kind  of  analysis  centers  on  a 
particular  event  (e.g.  an  airplane  crash  or  invention  of  the  printing  press).  It  attempts  to 
identify  the  events  or  causes  that  led  to  the  critical  event  or  the  set  of  consequences  that 
flow  from  the  critical  event. 

Cause-and-effect  analysis  is  a  variation  on  critical-event  analysis  that  assumes  a 
sequence  of  events,  each  one  leading  to  the  next.  It  is  frequently  used  in  constructing 
artificial  intelligence  models  of  events.  The  analysis  distinguishes  triggering  events  or 
causes  from  preconditions,  which  are  necessary  conditions  for  the  effects  to  occur.  Each 
effect  in  turn  can  be  the  triggering  event  for  a  new  set  of  effects. 
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Figure  2:  Figure  1  Military  analyst’s  beginning  analysis  of  water  rights 
problem 
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Problem-centered  analysis  is  found  throughout  history  and  any  subject  area 
where  human  goals  and  actions  are  paramount.  The  simplest  form  of  this  analysis 
breaks  an  event  stream  into  problems  and  actions  taken  to  solve  the  problems.  These 
actions  lead  to  main  effects  and  side  effects.  Problem-centered  analysis  is  embodied  in 
the  formal  analysis  of  human-computer  interaction  by  Card,  Moran,  and  Newell  (1983), 
and  the  analysis  of  electronic  troubleshooting  behavior  by  Hall,  Gott,  and  Pokorny 
(1992).  In  a  variation  on  the  Card,  Moran,  and  Newell  analysis,  the  latter  break  the 
stream  of  events  into  problems,  actions,  results,  and  interpretations  (called  PARI 
analysis).  Each  interpretation  identifies  a  new  problem  to  be  solved  unless  the  goal 
state  has  been  reached. 

Multi-factor  analysis  or  AND/OR  graphs  are  another  common  way  to  analyze 
causality  in  systems.  They  are  particularly  pervasive  in  geography  and  medicine,  but 
are  common  in  many  other  disciplines  where  it  is  difficult  to  identify  a  chain  of  events 
that  are  causally  interlinked.  In  multi-factor  theories,  variables  (called  factors  or 
independent  variables)  are  linked  together  in  a  tree  structure.  The  branches  of  the  tree 
are  ANDed  together  if  a  set  of  factors  are  all  necessary  to  produce  the  desired  value  on 
the  dependent  variable.  They  are  ORed  together  if  any  of  the  factors  are  sufficient  to 
produce  the  desired  value  on  the  dependent  variable. 

Form-and-function  analysis  involves  different  structures  depending  on  the  field 
of  inquiry.  The  simplest  analysis  is  to  distinguish  between  the  forms  of  objects  and 
their  functions  or  uses.  Weld  (1983)  used  a  more  sophisticated  form-and-function 
analysis  to  characterize  explanations  of  the  workings  of  physical  devices,  such  as  a  car 
engine  in  terms  of  roles,  functions,  structures,  and  mechanisms.  Another  artificial 
intelligence  researcher  (Edelson,  1993)  represented  the  knowledge  extracted  from 
interviewing  a  biology  professor  about  animal  behavior  in  terms  of  an  elaborated  form- 
and-function  analysis  specialized  for  biology. 

These  various  kinds  of  functional  or  causal  analysis  forms  guide  inquiry  by 
providing  target  structures  that  can  be  used  for  analyzing  phenomena  or  events  in  the 
world.  Such  functional  analysis  pervades  human  being's  attempts  to  make  sense  of  the 
world  around  them  (Perkins,  1986). 

Process-Analysis  Games 

In  addition  to  analyses  in  terms  of  structure  and  function,  a  third  kind  of  analysis 
of  phenomena  is  in  terms  of  their  dynamic  behavior.  We  have  labeled  the  various 
epistemic  forms  and  games  designed  to  make  sense  of  dynamic  phenomena  as  process 
analyses.  The  major  forms  and  games  we  will  discuss  are  system-dynamics  models, 
aggregate-behavior  models,  constraint-system  analyses,  situation-action  models,  and 
trend  analysis. 
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System-dynamics  models  are  increasingly  common,  especially  in  the  social  and 
physical  sciences.  There  are  a  variety  of  computer  programs,  such  as  STELLA,  that 
provide  tools  for  constructing  system-dynamics  models.  Such  programs  in  fact  provide 
a  generative  epistemic  form  for  playing  the  epistemic  game  of  system-dynamics 
modeling.  The  basic  elements  in  system-dynamics  models  are  variables  that  can 
increase  or  decrease.  These  are  linked  together  by  positive  or  negative  links,  usually 
with  feedback  loops  permeating  the  system  of  variables. 

Aggregate-behavior  models  are  constructed  frequently  to  explain  behavior  in 
the  physical  sciences,  particularly  the  behavior  of  small  particles  like  molecules  and 
electrons.  The  models  assume  random,  parallel  motion  of  a  large  array  of  particles. 
When  the  particles  encounter  each  other,  there  are  a  number  of  possible  interactions, 
such  as  sticking  together,  rebounding,  or  breaking  apart,  that  occur  under  different 
conditions.  When  they  encounter  a  barrier,  there  also  are  a  set  of  possible  interactions, 
such  as  penetrating  it,  rebounding  from  it,  or  sticking  to  it,  under  different  conditions. 
These  kinds  of  models  are  characteristic  of  diffusion  models,  chemical  mixtures, 
statistical  mechanics,  origin-of-life  models,  and  DNA  replication. 

Constraint  systems  have  permeated  our  understanding  of  physical  systems  since 
Galileo.  They  are  characterized  by  a  set  of  equations  describing  the  steady  state 
behavior  of  a  system.  Constraint  systems  are  regarded  as  one  of  the  most  precise  forms 
in  which  to  state  a  theory.  The  epistemic  game  most  associated  with  this  epistemic  form 
is  the  controlling-variables  game  developed  by  Galileo,  where  one  tries  to  manipulate 
one  variable  at  a  time  while  holding  other  variables  constant,  in  order  to  determine  the 
effect  of  each  independent  variable  on  the  dependent  variable. 

Situation-action  (production-rule)  models  are  commonly  used  to  describe 
behavior  in  the  social  sciences  (e.g.  Newell  &  Simon,  1972).  They  are  characterized  by  a 
set  of  rules  of  the  form  "If  in  situation  x,  do  y."  The  situation  can  change  either  because 
the  world  changes  or  because  the  agent  takes  an  action.  Markov  models  and  grammars 
can  be  considered  as  special  cases  of  situation-action  models,  where  each  action  takes 
one  into  a  new  state  and  where  different  actions  are  possible  in  each  state. 

Trend  analysis  is  most  commonly  found  in  economics  and  history,  but  can  be 
used  to  analyze  any  set  of  variables  that  change  over  time.  A  classic  example  of  this 
kind  of  analysis  is  Milankovitch's  analysis  of  how  the  earth's  ice  ages  depend  on  cyclic 
variations  of  the  earth's  tilt,  the  shape  of  the  earth's  orbit,  and  the  precession  of  the 
orbit.  Milankovitch  showed  how  precise  predictions  can  be  made  from  such  a  model. 

Process  analyses  attempt  to  characterize  the  behavior  of  dynamic  systems.  The 
process-analysis  games  we  have  identified  are  very  diverse,  and  appear  to  share  fewer 
characteristics  than  the  structural-analysis  and  functional-analysis  games.  The 
groupings  of  games  is  put  forward  as  our  best  hypothesis  as  to  which  games  share  the 
most  constraints. 

We  have  briefly  described  some  of  the  most  common  epistemic  games  and  the 
epistemic  forms  that  are  associated  with  them.  This  is  not  a  complete  list,  but  it  does 
serve  to  illustrate  the  potential  of  the  concept  of  epistemic  forms  and  games.  Mastering 
any  of  these  games  gives  one  a  powerful  tool  for  making  sense  of  different  phenomena 
in  the  world. 
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Protocol  Experiments  on  Expert  Analysis 


Method 

In  our  current  ARI  contract  we  have  collected  and  analyzed  protocols  from 
expert  military  analysts  and  scientists.  In  order  to  collect  protocols  from  military 
analysts,  we  visited  the  War  College  in  Carlisle  Pennsylvania  to  obtain  military 
scenarios  they  use  as  training  exercises.  We  obtained  a  number  of  scenarios  they  had 
developed  and  have  collected  protocol  data,  using  three  military  experts  as  subjects,  on 
two  of  these  scenarios.  One  of  the  scenarios  concerns  a  confrontation  over  water  rights 
between  Turkey,  Syria  and  Iraq  and  the  other  concerns  piracy  in  the  Straits  of  Malacca 
near  Singapore.  We  identified  eight  stopping  points  in  the  first  scenario  and  five  in  the 
second,  where  the  subjects  were  asked  "to  analyze  as  thoroughly  as  possible  what  the 
situation  is  and  to  make  a  detailed  prediction  as  to  what  you  think  will  happen  next." 
For  the  first  scenario,  a  former  Colonel  in  the  Army  worked  alone,  and  a  former 
Lieutenant  General  in  the  Army  and  a  former  Colonel  in  the  Marines  worked  jointly. 
For  the  second  scenario  all  three  worked  together.  All  the  sessions  lasted  less  than  two 
hours. 

The  science  protocols  were  collected  from  a  physicist,  biologist,  psychologist,  and 
historian,  who  worked  individually  on  four  questions  for  two  to  three  hours  each.  The 
four  questions  covered  a  range  of  topics  in  the  biological,  physical,  and  social  sciences, 
including  history,  and  relevant  materials  were  provided.  The  questions  were  meant  to 
require  different  kinds  of  solutions,  such  as  predictions,  explanations,  policy  decisions, 
and  analysis  of  patterns  in  data.  The  four  questions  were: 

•  What  is  the  cause  of  declining  wages  in  America  and  what  should  we  do  to 
reverse  the  decline?  The  materials  consisted  of  five  books  with  different  views  and 
explanations,  ranging  from  economic  to  political  to  demographic. 

•  How  will  the  climate  of  the  Earth  change  over  the  next  10  to  10,000  years?  Please 
project  the  changes  over  the  entire  period.  The  materials  consisted  of  about  20 
articles  about  climate  changes  discussing  various  change  agents,  such  as  carbon 
dioxide,  dust  particles,  orbital  variation,  sunspot  variation,  etc. 

•  How  do  teenagers  in  different  countries  differ?  Subjects  were  given  a  book  titled 
The  Teenage  World  with  a  large  data  set  from  ten  countries  of  teenagers'  answers  to 
about  TOO  psychosocial  true/ false  questions,  separated  by  sex  and  age.  The  text 
described  patterns  the  researchers  identified  in  the  data. 

•  What  do  you  predict  for  the  world's  resources  and  environment  over  the  next  50 
years?  Make  explicit  predictions  at  least  about  food,  energy,  materials,  pollution, 
and  wildlife.  The  materials  consisted  of  several  books  and  articles  giving  trend 
data,  explanations  for  the  trends,  and  future  projections. 
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We  counterbalanced  the  order  of  the  problems  across  subjects  and  asked 
them  to  work  according  to  the  following  instructions: 

1.  Your  task  is  to  analyze  the  problem  given  and  come  up  with  your  own  solution 
or  theory  to  address  the  problem. 

2.  When  you  are  finished  analyzing  the  problem,  we  want  you  to  write  up  your 
solution  in  some  form,  such  as  text,  a  figure,  a  table,  or  preferably  a  combination 
of  these. 

3.  You  will  have  2-3  hours  to  work  or  longer  if  you  need  the  time,  but  you  should 
try  to  finish  in  2  hours. 

4.  Spend  as  little  time  as  possible  reading.  Try  to  work  through  the  problem  using 
the  materials  as  a  reference. 

5.  Your  job  is  to  construct  your  own  analysis  of  the  problem.  Do  not  give  us  the 
solutions  or  theories  of  the  authors  you  are  reading. 

6.  Bring  in  whatever  knowledge  you  have  from  other  sources.  Try  to  identify  what 
that  knowledge  is  and  its  source. 

7.  Try  to  make  integrative  notes  and  representations  of  your  thinking  as  you  go 
along.  The  more  detailed  your  notes,  the  better. 

8.  Explain  your  thinking  as  you  go  as  thoroughly  as  possible  into  the  tape  recorder. 
You  can  pause  the  tape  recorder  while  you  read,  so  we  do  not  get  a  lot  of  blank 
tape. 

9.  Try  to  write  down  and  say  what  your  current  solution  is  as  you  go  along, 
explaining  why  you  think  what  you  do  and  why  you  make  any  changes  in  your 
view. 

10.  If  you  change  your  mind  about  something,  make  a  special  effort  to  make  note  of 
the  change  and  what  caused  it.  It  is  easy  to  not  notice  you  are  changing  your 
mind,  so  please  make  a  special  effort  to  notice  any  change. 

11.  When  you  start  generating  a  particular  representation,  try  to  say  where  it  came 
from. 

12.  Do  not  feel  you  need  to  read  or  even  look  at  all  the  materials.  Select  those  that 
interest  you  most  and  focus  your  investigation  however  you  think  is  most 
productive. 

Analysis  of  the  Protocols 

In  our  analysis  we  have  identified  six  different  kinds  of  elements  that  the  experts 
are  using  to  analyze  these  complex  problems:  epistemic  forms  (target  structures  for  the 
analysis),  epistemic  games  (analysis  strategies),  domain  frameworks  (organizing 
structures  for  guiding  an  analysis),  key  constructs  (key  variables  that  enter  into 
different  types  of  models),  heuristics  (rules  of  thumb,  such  as  to  look  for  positive 
feedback  loops  or  lag  effects),  and  prompts  (issues  raised  for  consideration  by  the 
experts).  Below  we  discuss  each  of  these  and  the  role  they  have  played  in  the  protocols. 
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Epistemic  Forms.  Epistemic  forms  are  the  target  structures  that  guide  the 
experts  in  their  analysis.  There  is  a  progression  of  forms  we  have  noted  in  the 
protocols,  where  subjects  nest  one  kind  of  analysis  inside  another  kind  of  analysis.  For 
example,  they  often  used  a  structural-analysis  form,  such  as  a  list  or  stage  model;  in 
order  to  make  a  decomposition  of  the  problem.  Then  within  each  component,  they 
used  another  analysis  form,  such  as  a  trend  analysis  or  a  cause-and-effect  analysis,  to 
elaborate  their  analysis.  As  another  example  of  how  they  nest  their  analyses,  they 
would  sometimes  start  working  out  causal  chains,  which  they  then  would  develop  into 
multi-factor  models.  In  one  case  the  biologist  extended  his  multi-factor  model  into  a 
system-dynamics  model.  Thus  the  analyses  they  constructed  involved  complex  nesting 
of  different  epistemic  forms. 

There  are  ten  epistemic  forms  that  we  identified  in  the  protocols: 

•  Cause-and-effect  analysis.  Both  the  military  analysts  and  the  scientists  spent  a  large 
amount  of  time  constructing  causal  chains.  The  causal  chains  constructed  had  the 
following  form:  S(a)  =>  S(b)  =>  S(c),  where  S  is  a  state.  In  the  military  protocols,  the 
links  were  often  assigned  qualitative  likelihoods.  The  chains  were  of  different 
lengths,  but  most  contained  two  or  three  links.  An  example  of  this  kind  of  analysis 
for  the  water-rights  scenario  is  shown  below: 

•  Multi-factor  models.  In  several  of  the  problems  the  experts  attempted  to  create  a 
multi-factor  causal  model  in  the  form  of  an  AND/OR  graph.  As  an  example  of  this 
kind  of  analysis  created  to  account  for  why  real  wages  have  declined  in  America 
during  recent  year,  one  scientist  constructed  a  multi-factor  model,  where  the  key 
constructs  were  labor  supply,  labor  demand,  technology,  global  competition  for 
jobs,  the  baby  boom  entering  the  workforce,  women  entering  the  workforce, 
increasing  costs  of  benefits,  and  trade  policies.  These  factors  were  linked  together  to 
account  for  the  declining  wages,  based  on  a  supply-and-demand  framework. 

•  Cost-benefit  analysis.  In  the  military  protocols,  the  subjects  often  considered 
different  possible  courses  of  action  and  the  costs  and  benefits  associated  with  each. 
In  one  case  they  considered  three  possible  courses  of  action  with  respect  to  dealing 
with  pirates  in  the  Straits  of  Malacca:  (1)  Putting  pressure  on  Indonesia  to  deal  with 
the  pirates,  (2)  taking  unilateral  action  against  the  pirates,  and  (3)  organizing  a 
multilateral  force  against  the  pirates.  We  can  illustrate  the  kinds  of  costs  and 
benefits  they  considered  in  terms  of  multilateral  action.  The  benefits  they  considered 
for  multilateral  action  were  spreading  costs  over  more  parties,  adding  legitimacy, 
and  building  relationships  with  countries  in  the  region.  The  costs  involved  the 
difficulty  in  taking  decisive  action  and  the  extensive  negotiation  and  time  it  takes  to 
put  together  multilateral  action.  The  analysts  updated  their  cost-benefit  analysis  as 
the  situation  unfolded  in  the  scenario.  There  are  classes  of  costs  and  benefits  (e.g., 
financial,  international  relations)  that  we  have  begun  to  enumerate  based  on  our 
analyses.  While  the  problems  given  the  scientists  did  not  lead  them  to  use  cost- 
benefit  analysis,  an  earlier  protocol  we  collected  from  a  scientist,  as  to  how  New 
York  City  should  dispose  of  its  garbage,  involved  an  extensive  cost-benefit  analysis. 
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•  Comparative  analysis.  A  variation  on  cost-benefit  analysis  occurred  in  analyzing 
the  piracy  problem  when  two  of  the  analysts  engaged  in  an  argument  as  to  whether 
Indonesia  or  China  might  be  helping  the  pirates,  since  they  were  operating  down 
near  Singapore.  This  led  to  a  comparison  of  arguments  for  and  against  China  and 
Indonesia,  as  below: 


China 


Arguments  For 

Reports  of  talks  w/pirates 
Diversion  from  Spratleys, 
Where  China  seeks  oil 


Arguments  Against 

Not  clear  what  they  would  gain 
Cannot  patrol  the  Straits 


Indonesia 


Their  navy  ships  involved  Assoc  with  rebellion  in  Sumatra 

Pirates  in  waters  near  Java  Pirate  bases  in  Nicobar  Islands 


•  Trends.  Both  on  the  climate  problem  and  the  resources  problem  the  scientists 
produced  trend  analyses,  based  on  their  understanding  of  the  types  of  functions 
(e.g.,  exponential  curves,  normal  growth  functions)  that  characterize  different 
processes.  For  example,  on  the  climate  problem  the  psychologist  started  identifying 
key  variables  and  their  effects  on  climate  trends.  When  he  came  to  plotting  his 
predictions,  he  ended  up  with  three  phases.  The  first  was  an  exponential  increasing 
function  caused  by  the  increase  of  greenhouse  gases,  partially  offset  by  orbital 
variation  and  pollutants.  The  second  phase  was  a  decreasing  function  down  to  a 
level  below  current  temperatures,  caused  by  orbital  variation  as  people  reduce 
greenhouse  gases.  The  third  phase  showed  an  increasing  asymptotic  function  up  to 
some  optimal  level  near  current  temperatures,  as  humans  learn  to  control 
temperature  through  greenhouse  emissions  or  other  mechanisms.  His  predictions 
combined  two  of  the  epistemic  forms:  stage  models  and  trend  analysis. 

•  Stage  or  phase  models.  The  military  analysts  created  phase  models  to  characterize 
how  confrontation  processes  unfold  over  time.  This  arose  in  the  analysis  of  the 
water-rights  problem  where  the  analyst  decided  that  Syrian  military  moves  were  for 
show,  because  the  confrontation  was  still  in  the  negotiation  phase.  One  such  stage 
model  functioned  as  a  domain  framework  that  organized  the  entire  analysis,  as 
described  below.  We  saw  an  example  of  how  scientists  constructed  stage  models  in 
the  example  above. 

•  Lists.  In  the  protocols,  the  subjects  frequently  used  a  list  analysis  to  decompose  the 
problem  into  components.  For  example,  one  subject  in  predicting  the  fate  of 
different  species,  broke  them  into  three  types:  animals  that  humans  would  save  (e.g., 
gorillas),  animals  that  thrive  in  conjunction  with  humans,  and  animals  that  would 
likely  become  extinct. 

•  Spatial  decomposition.  In  a  variant  on  list  making,  the  military  analysts  on  the 
piracy  problem  went  through  each  of  the  countries  in  the  area  to  determine  their 
goals  and  how  they  might  be  encouraged  to  act  to  stop  the  piracy. 
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•  Finite  state  models.  The  physicist  in  working  on  the  wages  problem  created  a 
simple  two-state  model,  which  is  a  special  case  of  a  finite-state  model.  The  two 
stable  states  he  depicted  were:  1)  high  wages,  women  not  working,  family  income  = 
x,  and  2)  low  wages,  women  working,  family  income  =  x',  where  x  and  x'  are  not 
very  different.  He  characterized  recent  decades  as  moving  from  state  1  to  state  2, 
brought  on  by  World  War  II  and  the  women's  movement. 

•  System-dynamics  model.  The  biologist  in  working  on  the  resources  problem 
created  a  system-dynamics  model,  where  he  tried  to  incorporate  all  five  of  the 
variables  (food  supply,  energy,  materials,  wildlife  and  pollution)  specified  in  the 
problem  with  six  other  variables  (e.g.,  climate,  technology,  policies)  in  a  complex 
interaction  pattern,  with  multiple  feedback  loops. 

Epistemic  Games.  In  the  Collins  and  Ferguson  (1993)  paper  there  was  implied 
coupling  between  epistemic  forms  and  games,  such  that  there  was  one  game  for  every 
form.  The  protocols  show  that  while  there  are  epistemic  games  associated  with  each 
form,  some  analysis  strategies  are  not  associated  with  specific  epistemic  forms.  The 
multi-purpose  analysis  strategies  that  we  have  identified  in  the  protocols  are: 

•  Hypothesis  formation  and  testing.  Subjects  often  formulated  hypotheses  about 
what  the  situation  might  be  or  what  might  happen.  For  example,  one  military 
expert  hypothesized  that  Syria  and  Iraq  might  attack  Turkey  in  the  water-rights 
situation.  Then  he  went  through  the  scenario  as  it  unfolded  looking  for  evidence 
that  Syria  and  Iraq  were  planning  an  attack. 

•  Looking  for  and  explaining  anomalies.  The  historian  used  a  variation  on  the 
hypothesis  testing  strategy  that  is  worth  noting.  In  the  teenage  problem,  she  would 
test  different  hypotheses  looking  for  cases  (either  properties  or  types)  that  did  not  fit 
the  overall  pattern.  Then  she  would  try  to  construct  a  causal  chain  that  would 
account  for  the  anomaly.  The  military  experts  in  the  piracy  scenario  used  a  similar 
strategy  in  trying  to  explain  an  anomaly  about  why  there  was  an  attack  in  the  Straits 
of  Malacca  near  Singapore,  when  the  pirates  were  supposedly  based  in  the  Nicobar 
Islands.  They  suspected  that  the  pirates  must  therefore  be  receiving  help  from  some 
country  in  the  region. 

•  Identifying  factors  or  key  constructs.  Much  of  the  thinking  and  reading  that 
subjects  did  was  directed  toward  identifying  factors  that  they  think  may  have  an 
influence  on  the  dependent  variable  in  question.  For  example,  in  the  piracy  scenario 
different  experts  raised  the  issues  of  multilateral  action,  the  legality  of  the  US  to 
send  in  forces,  whether  the  Straits  were  international  waters,  what  countries  would 
be  affected  by  the  piracy,  etc.  Also  on  the  wages  problem  the  scientists  tried  to 
identify  factors,  such  as  women  entering  the  work  force,  that  might  tend  to  drive 
down  wages.  These  key  constructs  entered  into  all  the  different  types  of  epistemic 
forms  they  created. 
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•  Constructing  causal  chains.  Both  military  experts  and  scientists  often  construct 
casual  chains  by  which  one  event  or  variable  affects  another  event  or  variable.  The 
military  experts  sometimes  assigned  qualitative  likelihoods  to  the  causal  links.  In 
addition  to  creating  cause-and-effect  models,  the  causal  chains  also  support 
construction  of  multi-factor  models  and  a  strategy  of  countering  the  weakest  link 
(described  below). 

•  Countering  the  weak  links.  Military  analysts  often  constructed  ways  to  prevent 
bad  consequences  they  foresaw.  Similarly  the  scientists  were  asked  to  propose 
solutions  to  the  problem  of  declining  wages.  There  were  two  counterplanning 
strategies  used  for  addressing  such  issues.  The  main  strategy  was  based  on 
identifying  weak  links  in  a  causal  chain  and  then  figuring  out  a  way  to  counter  those 
links.  For  example,  in  the  water-rights  problem  the  goal  was  to  avoid  a  military 
confrontation,  so  one  suggestion  was  to  provide  mediation  between  the  parties  to 
the  dispute. 

•  Countering  from  strength.  The  other  counterplanning  strategy  used  was  to  start 
with  something  you  know  how  to  do  well  and  applying  that  to  the  problem  at  hand. 
So  for  example,  two  of  the  scientists  attacked  the  problem  of  declining  wages  by 
specifying  different  ways  that  education  might  address  the  problem. 

Domain  Frameworks.  It  was  striking  that  both  military  experts  and  scientists 
used  general  frameworks  that  organized  both  large  and  small  portions  of  their  inquiry. 
We  have  labeled  these  domain  frameworks,  because  unlike  epistemic  forms  and  games 
they  are  full  of  domain-specific  content.  In  fact  they  are  instantiations  of  particular 
epistemic  forms.  For  example,  one  military  analyst  used  an  instantiated  stage  model 
(described  below)  to  guide  his  analysis  in  the  water-rights  scenario.  There  was 
abundant  use  of  analogies  in  the  protocols,  and  we  think  they  are  functioning  as 
domain  frameworks.  We  will  give  four  examples  of  domain  frameworks  organizing  an 
inquiry,  but  there  are  many  more  contained  in  the  protocols. 

•  Confrontation  phase  model.  A  military  expert  interpreted  developments  in  the 
Turkey-Syria-Iraq  water  rights  problem  in  terms  of  a  three-phase  model  of 
confrontation.  In  the  first  phase,  the  negotiation  phase,  there  is  an  effort  by  the 
conflicting  parties  to  resolve  their  differences  peacefully  with  the  help  of  mediation. 
Any  military  maneuvers  that  occur  before  this  phase  is  completed  are  mainly  to 
show  serious  intent.  The  phase  ends  when  negotiations  fail,  which  leads  to  military 
preparation.  The  preparation  phase  consists  of  moving  troops  and  supplies  into 
position,  and  ends  in  military  attack.  At  any  time  during  the  preparation  or  attack 
phases,  the  process  can  return  to  the  negotiation  phase,  but  this  usually  does  not 
occur  until  a  stalemate  is  reached. 
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•  Law  of  supply  and  demand.  The  scientists  attacked  the  wages  problem  using  the 
law  of  supply  and  demand  to  organize  their  search  for  factors  that  might  cause  a 
decline  in  wages.  To  do  this  they  looked  for  things  that  might  increase  the  supply  of 
labor  or  decrease  the  demand  for  labor  or  for  goods  produced  by  the  labor.  Supply 
and  demand  is  a  complex  concept  with  many  facets  and  it  clearly  played  a  large  role 
in  the  inquiry.  But  it  is  an  instantiated  theory  rather  than  an  epistemic  form. 

•  Frontier  metaphor.  The  historian  in  working  on  the  resources  problem  invoked  the 
parallel  to  the  American  frontier,  where  people  would  move  on  to  new  territories 
when  all  the  land  was  taken.  She  came  to  see  resources  as  like  the  frontier  and  so 
organized  her  inquiry  around  the  way  that  people  would  move  on  to  other 
resources  when  the  old  ones  were  used  up. 

•  Principle  of  attacking  when  your  opponent  is  diverted.  One  military  expert 
applied  a  general  rule  of  aggressive  behavior  by  countries  in  trying  to  predict  what 
Iran  might  do  in  the  water-supply  problem.  The  framework  he  used  had  the  general 
form  that  if  your  opponent  (e.g.,  Iraq)  is  occupied  with  a  problem  (e.g.,  Turkey), 
then  this  is  an  opportune  time  to  move  against  your  opponent.  These  kinds  of 
domain-inference  frameworks  were  used  quite  frequently  to  make  predictions  in  the 
military  protocols. 

•  Lester  Brown  vs.  Julian  Simon  model  of  resources.  In  working  on  the  resource 
problem,  the  psychologist  started  his  analysis  based  on  one  of  the  readings  with  a 
domain  framework  we  call  the  Lester  Brown  model  of  resource  depletion.  In  this 
view,  land  for  growing  crops  is  being  lost  to  soil  erosion,  salination,  and  other  uses, 
natural  resources  are  finite  and  are  being  used  up,  and  we  are  reaching  the  point  of 
diminishing  ability  to  increase  resource  production  to  accommodate  increasing 
population.  But  after  pursuing  this  framework  for  a  short  while,  he  switched  to 
another  framework  we  call  the  Julian  Simon  model  of  infinite  resources,  based  on 
one  of  the  readings  among  the  materials  provided.  He  used  the  Simon  model 
thereafter  in  making  predictions  about  food,  energy,  and  mineral  resources. 
According  to  the  Simon  model  the  cost  of  resources  over  the  long  run  always 
decreases,  because  of  substitution,  improved  production  techniques,  and  opening  up 
of  new  sources.  The  decreasing  cost  reflects  increasing  supply  relative  to  demand. 

Key  Constructs.  All  the  subjects  identified  key  constructs  that  they  worked  from 
in  building  causal  chains,  multi-factor  models,  and  in  one  case  a  system-dynamics 
model.  These  are  concepts  like  the  Munich  pact  in  history  or  money  supply  in 
economics.  They  function  as  the  building  blocks  in  any  type  of  theory  the  subjects 
develop.  We  will  give  four  examples  of  key  constructs  different  subjects  used. 
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•  Islamic  War.  One  military  analyst  invoked  a  notion  of  an  Islamic  War  with  the  West 
as  a  dangerous  possibility  that  actions  should  be  taken  to  avoid.  This  notion  of  an 
Islamic  War  invokes  the  notion  that  a  number  of  Islamic  countries  get  together,  as 
they  have  in  the  past  to  attack  Israel,  to  fight  attempts  by  Western  powers  to  control 
their  behavior  or  impose  their  culture.  The  military  experts  invoked  a  number  of 
such  scenarios  (e.g.,  a  cutoff  of  oil  by  Islamic  countries,  a  Greek-Turkish 
confrontation)  in  order  to  infer  what  different  parties  are  likely  to  do  in  a  given 
situation. 

•  Labor  supply,  etc.  The  scientists  referred  to  the  key  variables  of  labor  supply,  labor 
demand,  supply  of  goods,  and  demand  for  goods,  in  working  through  the  declining 
wages  problem.  These  are  variables  that  come  out  of  the  supply-and-demand 
framework  that  subjects  used  to  organize  their  analysis  of  the  problem.  These  are 
what  are  referred  to  in  the  psychological  literature  as  "intermediate  constructs." 

•  Negotiations,  military  preparation,  attack.  The  key  constructs  in  the  military 
protocols  included  the  different  phases  that  military  confrontations  go  through. 
These  constructs  emerged  out  of  the  domain  framework  that  the  military  experts 
used,  just  as  variables,  such  as  labor  supply  and  demand  for  goods,  arose  out  of  the 
supply-and-demand  framework  that  the  scientists  used. 

•  Greenhouse  effect,  etc.  All  of  the  scientists,  as  they  worked  on  the  climate  problem, 
identified  a  number  of  controlling  variables.  The  greenhouse  effect  and  orbital 
variation  were  the  largest  of  these  variables.  One  subject  saw  his  first  task  as 
identifying  all  of  the  key  variables  that  affect  climate,  and  then  determining  which 
had  the  greatest  effect. 

Heuristics.  Often  subjects  talked  about  processes  and  effects  that  were  part  of 
their  toolkit  for  understanding  the  phenomena  that  they  were  dealing  with,  but  that 
were  less  elaborated  than  the  domain  frameworks  and  more  content  specific  than  the 
epistemic  forms.  We  call  these  heuristics. 

•  Time  course.  The  military  experts  referred  to  the  time  course  over  which  different 
phases  of  events  occur.  For  example,  negotiations  or  deploying  warships  to  protect 
merchant  ships  from  pirates,  have  a  particular  time  course  to  have  effect.  They  used 
these  time  estimates  to  interpret  events  that  were  happening. 

•  Lag  effects.  Several  of  the  subjects  talked  about  lag  effects  or  time  delays  in 
problems  where  predictions  and  remedies  were  called  for.  So  for  example  in  the 
climate  problem  the  scientists  talk  about  the  time  delays  between  when  humans  do 
something  (e.g.,  releasing  carbon  dioxide)  and  its  full  effects  are  felt  (e.g.,  earth 
warming).  They  arose  in  the  military  protocols  in  terms  of  how  long  it  would  take 
to  put  a  coalition  together  or  to  move  troops  into  position.  Lag  effects  then  are  used 
to  explain  certain  kinds  of  system  dynamics. 
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•  Feedback  loops.  There  are  two  kinds  of  feedback  loops  that  the  subjects  used  in 
making  sense  of  the  problems,  positive  and  negative  feedback  loops.  In  the  military 
scenarios,  feedback  loops  occur  when  countries  take  actions  that  lead  to  escalating 
actions  by  adversaries.  The  scientists  also  described  feedback  loops,  such  as  when 
the  physicist  noted  that  snow  leads  to  increased  reflection  of  heat  into  the 
atmosphere,  causing  further  cooling. 

•  Likelihood  of  effects.  In  the  military  protocols  the  subjects  often  referred  to  the 
likelihood  that  certain  events  would  lead  to  other  events.  This  sometimes  occurred 
in  the  science  protocols,  when  human  actions  were  invoked  to  explain  what  might 
happen  in  the  climate,  wages ,  and  resources  problems. 

•  Size  of  effects.  Another  important  notion  that  came  up  in  several  of  the  science 
protocols  is  the  relative  size  of  different  effects.  This  idea  was  used  at  least  by  the 
psychologist  to  discount  certain  factors  that  he  did  not  think  would  affect  a 
dependent  variable  as  much  as  other  factors.  In  fact  he  spent  a  fair  amount  of  time 
on  the  climate  problem  trying  to  determine  the  relative  size  of  effects  of  carbon 
dioxide  and  orbital  variation. 

•  Positive  vs.  negative  consequences.  The  military  analysts  made  an  important 
distinction  between  positive  consequences  and  negative  consequences,  always  with 
respect  to  a  particular  party,  such  as  Turkey  or  NATO.  The  assumption  was  that 
parties  would  take  actions  intended  to  produce  positive  consequences  and  would 
avoid  actions  likely  to  lead  to  negative  consequences. 

•  Side  effects.  Finally  several  of  the  subjects  described  certain  effects  as  side  effects, 
implying  that  they  were  not  intended  by  the  parties  taking  an  action. 

Prompts.  There  was  a  strategy  that  arose  in  the  military  protocols,  but  not  in  the 
scientist's  protocols,  which  depended  on  the  interaction  between  experts.  We  have 
labeled  these  "prompts,"  because  one  of  the  experts  would  bring  up  an  issue  for  the 
group  to  consider.  Any  tool  we  design  to  support  expert  analysis  would  naturally  have 
a  set  of  prompts  that  it  might  use  to  provoke  the  experts  to  consider  different 
alternatives.  We  give  examples  of  the  prompts  that  occurred  in  the  military  protocols. 

•  Downsides.  In  considering  different  courses  of  action,  such  as  whether  to  send  US 
ships  to  the  Malacca  Straits,  one  of  the  experts  often  raised  the  question  of  what  the 
downsides  (or  costs)  would  be. 

•  Possible  Actions.  One  of  the  experts  raised  the  possibility  of  multilateral  action, 
after  they  had  been  considering  unilateral  action. 

•  Legality.  One  of  the  experts  often  raised  the  question  of  whether  a  particular  action 
was  legal  under  international  law,  such  as  Indonesia  collecting  fees  for  passage 
through  the  Straits  of  Malacca. 
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•  Countries  Affected.  One  of  the  prompts  raised  was  what  countries  would  be 
affected  by  any  particular  action. 

•  Deception.  A  particular  issue  that  was  raised  on  several  occasions  was  whether  an 
action  was  what  it  appeared  to  be,  or  whether  it  was  a  deception  of  some  kind,  such 
as  the  movement  of  troops  by  Iraq  and  Syria  in  the  water  rights  scenario. 

Conclusion.  In  the  protocols  collected  from  military  experts,  there  were  three 
different  levels  of  analysis  and  two  different  types  of  analysis  that  the  experts  used.  The 
three  levels  of  analysis  were  concerned  with  different  actors:  (1)  individuals  and  groups 
(e.g.,  the  foreign  minister  of  Turkey  or  the  pirates),  (2)  countries  (e.g.,  Syria  or 
Indonesia),  and  (3)  alliances  (e.g.,  UN,  NATO,  or  ASEAN).  The  two  types  of  analysis 
involved  (1)  interpretation  of  actor's  goals,  intentions,  and  possible  future  actions,  and 
(2)  considering  different  courses  of  action  with  respect  to  the  actors.  This  same 
distinction  between  interpretation  and  action  arose  in  the  wages  problem  given  to 
scientists,  where  they  were  asked  to  analyze  the  reason  for  the  decline  in  wages  and  to 
propose  remedies  for  it. 

The  military  protocols  suggest  a  number  of  things  for  the  design  of  a  tool  to 
support  expert  analysis.  Clearly  the  kinds  of  prompts  that  occurred  would  be  very 
useful,  and  each  epistemic  form  suggests  a  number  of  prompts  that  could  be  posed  to 
experts.  It  was  also  apparent  that  there  were  a  number  of  different  types  of  costs  and 
benefits  that  the  experts  considered  (e.g.,  financial,  international  relations)  and  these 
could  act  as  prompts  in  doing  a  cost-benefit  analysis  in  considering  different  courses  of 
action.  The  protocols  also  provide  a  rich  source  of  domain  frameworks  and  key 
constructs  that  can  be  built  into  such  a  tool. 
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Design  of  a  Tool  to  Support  Expert  Analysis 

Based  on  our  experiments  we  have  begun  to  design  a  tool  that  would  support 
analysts  as  they  work  on  problems  like  those  we  used  in  the  experiments.  We  see  the 
tool  as  functioning  interactively  with  analysts  as  they  try  to  interpret  a  situation,  and 
develop  courses  of  action  to  deal  with  the  situation.  We  will  outline  our  preliminary 
design  in  terms  of  how  the  tool  would  support  analysts  faced  with  the  military 
scenarios  that  we  used  in  the  experiments. 

The  tool  we  envision  would  prompt  analysts  to  consider  all  the  important  actors 
in  a  situation,  the  stages  of  the  developing  action,  the  possible  interpretations  of  actions 
by  others,  the  possible  courses  of  action,  and  the  costs  and  benefits  of  different  courses 
of  action.  Analysts  may  respond  or  not  respond  to  any  of  the  prompts.  Each  prompt 
they  respond  to  may  lead  to  further  prompts  in  order  to  fill  out  the  corresponding 
epistemic  form. 

The  prompts  support  the  construction  of  models  of  the  situation.  We  see  these 
models  as  comprising  two  aspects  of  the  situation:  1)  an  interpretive  analysis,  where  the 
goal  is  to  determine  the  intentions  of  the  different  actors,  and  2)  an  action  analysis, 
where  the  goal  is  to  consider  different  courses  of  action  and  their  corresponding  costs 
and  benefits.  The  tool  will  support  analysts  to  update  their  models  as  the  situation  or 
their  understanding  unfolds. 

Interpretive  Analysis 

In  order  to  prompt  analysts  to  consider  all  the  important  actors  in  a  situation  we 
would  prompt  them  to  respond  at  the  three  levels  identified  in  the  protocols:  1)  the 
individual  and  group  level  (Assad  in  Syria  or  the  pirates  in  the  Malaccan  Straits),  2)  the 
country  level  (Turkey  or  Indonesia),  and  3)  the  institutional  level  (NATO  or  the  UN). 
To  identify  the  relevant  countries  and  institutions,  the  system  would  prompt  the 
analysts  to  consider  all  the  countries  and  institutions  in  the  region,  where  the  situation 
is  developing.  For  example,  while  the  analysts  in  the  pirate  scenario  considered 
Indonesia,  Singapore,  Malaysia,  and  China,  they  did  not  consider  Thailand  or  Myanmar 
(Burma).  This  uses  a  spatial  decomposition  strategy  to  guide  the  analysis. 

To  guide  the  interpretation  of  the  situation  the  system  would  prompt  the 
analysts  about  the  stage  of  development  of  the  situation  and  the  relations  between  the 
participants.  In  the  water-rights  scenario  the  expert  analysts  based  their  interpretation 
on  a  three-stage  model:  1)  negotiation  between  the  conflicting  parties,  2)  military 
preparations,  and  3)  military  conflict.  When  they  saw  troop  movements  before  the 
negotiations  had  run  their  course,  they  interpreted  them  as  designed  to  show 
seriousness,  rather  than  as  plans  to  take  military  action.  This  three-stage  model  in 
generalized  form  can  provide  a  basis  for  prompting  analysts  in  many  situations.  For 
example,  in  conflict  situations  the  system  can  prompt  the  analysts  to  specify  the  stage  of 
the  conflict,  their  certainty  with  regard  to  that  choice,  and  how  they  would  interpret 
each  action  in  light  of  that  stage. 
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In  the  piracy  scenario  the  experts  did  not  consider  negotiations  with  the  pirates, 
probably  because  they  are  like  terrorists  who  are  not  amenable  to  negotiation.  So  they 
jumped  to  considering  courses  of  action  in  preparation  for  military  action.  The  system 
would  have  prompted  them  to  consider  whether  negotiations  with  the  pirates  were 
possible. 

Another  kind  of  interpretive  prompt  would  support  analysts  in  constructing 
causal  chains  such  as  that  shown  above  in  Figure  2  depicting  one  expert's  analysis  of 
what  could  happen  in  the  water-rights  scenario.  The  system  would  prompt  the  analysts 
to  specify  different  actions  participants  might  take  at  any  point  and  the  likelihood  of 
that  choice.  It  would  then  ask  what  actions  other  parties  are  likely  to  take  in  response. 
The  system  would  ask  the  experts  to  decide  for  multiple  actions,  whether  their 
projections  are  ORed  together  (where  likelihoods  should  add  to  1)  or  are  ANDed 
together  (where  likelihoods  are  independent).  This  would  enable  the  analysts  to  project 
possible  scenarios  out  to  as  much  depth  as  they  feel  is  useful.  The  system  can  also 
prompt  the  analysts  to  project  backwards  to  the  precursors  for  actions  taken,  if  that  is 
helpful  to  them.  Based  on  the  expert's  inputs,  the  system  would  automatically 
construct  a  diagram  for  causal  analyses,  such  as  shown  in  Figure  2. 

In  the  protocols,  the  analysts  raised  the  question  a  number  of  times  as  to  whether 
the  actions  taken  were  what  they  appeared  to  be,  or  whether  there  was  some  deception 
going  on.  So  for  example,  one  of  the  analysts  in  the  piracy  scenario  raised  the  question 
of  whether  the  Chinese  might  be  supporting  the  pirates  to  provide  a  diversion  from  the 
Chinese  attempt  to  seize  the  Spratley  Islands.  We  propose  that  the  system  prompt 
analysts  to  consider  whether  there  is  another  possible  interpretation  involving 
deception  for  any  important  event  that  occurs. 

Action  Analysis 

In  order  to  support  the  analysts  in  deciding  upon  courses  of  action,  we  plan  to 
encourage  them  to  set  up  tables  of  the  possible  actions  and  their  costs  and  benefits.  An 
example  of  the  kind  of  table  we  envision  is  shown  below  in  Figure  3. 
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Figure  3.  Example  of  a  Cost-Benefit  Epistemic  Form 

In  order  to  fill  out  such  a  table  the  analysts  would  be  prompted  to  consider 
different  possible  courses  of  action.  The  possible  courses  of  action  for  the  negotiating 
phase  that  were  brought  up  in  the  protocols  included: 

•  Have  disputants  meet  together  and  discuss  their  differences. 

•  Provide  mediation  to  come  up  with  neutral,  face  saving  solutions. 

•  Provide  guarantees  or  resources  to  alleviate  the  concerns  of  the  disputants. 

•  Provide  pressure  if  they  fail  to  agree,  such  as  freezing  assets,  imposing  sanctions,  or 
military  pressure. 

The  courses  of  action  that  were  considered  in  the  military-preparation  phase 
included: 

•  Multilateral  action,  either  by  an  institution  (e.g.  NATO)  or  by  an  ad-hoc  coalition. 

•  Unilateral  action. 

•  Pressure  on  other  countries  to  take  action  (e.g.  on  Indonesia  to  stop  the  pirates). 

Neither  of  the  scenarios  involved  the  actual  planning  of  military  actions,  so  that 
the  respondents  did  not  develop  courses  of  action  for  the  military-action  phase,  but  the 
system  could  support  them  in  making  such  decisions  as  well. 

The  likelihood  of  success  came  up  in  the  expert's  discussion  of  different 
alternatives,  so  that  it  makes  sense  for  the  system  to  prompt  analysts  to  specify  their 
best  estimates  of  those  likelihoods.  In  some  cases  they  may  feel  that  the  likelihood  of 
partial  success  is  greater  than  the  likelihood  of  overall  success,  so  the  system  should 
allow  them  to  specify  likelihoods  in  more  complex  ways,  if  warranted. 


In  Figure  3  above  we  have  shown  several  of  the  different  kinds  of  costs  and  benefits 
the  experts  discussed  in  their  protocols,  which  the  system  would  use  as  prompts.  One 
of  the  analysts  kept  asking  the  group  to  consider  the  "downsides"  or  costs  of  any  action, 
because  he  knew  that  there  is  a  natural  human  tendency  to  consider  only  the  benefits, 
and  he  had  trained  himself  to  address  the  costs  as  well  as  benefits.  Any  type  of  potential 
benefit  is  also  a  type  of  potential  cost,  so  we  have  listed  the  same  set  for  both  in  Figure 
3.  We  briefly  describe  the  different  types  of  costs  and  benefits  in  the  Figure: 

•  Financial  benefits  refer  to  the  money  saved  or  the  resources  acquired,  if  an  action  is 
successful.  Financial  costs  refer  to  such  things  as  providing  resources  or  paying 
others  to  do  so. 

•  Military  benefits  refer  to  the  experience  gained  from  participating  in  an  action  and 
the  evaluation  of  new  weapon  systems  in  combat.  Military  costs  refer  to  the  loss  of 
personnel  and  resources. 

•  International-relations  benefits  refer  to  good  relations  that  are  developed  with 
countries  as  they  address  a  problem  together.  International-relations  costs  refer  to 
the  antagonism  from  other  countries  spawned  by  an  action.  Legality  was  central  to 
several  of  the  discussions,  and  it  could  be  treated  as  part  of  international  relations  or 
as  a  separate  category. 

•  Popular-support  benefits  refer  to  positive  reactions  from  the  population  of  a 
country  to  an  action  taken  by  that  country.  Popular-support  costs  refer  to  the 
corresponding  negative  reactions. 

•  Economic  benefits  refer  to  positive  effects  on  the  economy  of  a  country  resulting 
from  an  action.  Economic  costs  refer  to  negative  effects  on  the  economy. 

There  are  other  possible  types  of  costs  and  benefits  that  may  arise  in  other 
scenarios,  so  that  we  would  have  to  study  experts  working  with  other  problems  to 
determine  the  best  set  of  prompts  to  use. 
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Conclusion 


Our  studies  of  expert  analyses  have  revealed  many  important  aspects  of  the 
process  by  which  experts  create  models  of  a  situation,  beyond  our  preliminary  typology 
of  epistemic  forms  and  games.  While  epistemic  forms  and  games  did  turn  out  to  be 
critical  aspects  of  the  analysis  process,  we  have  identified  a  number  of  other  critical 
components,  i.e.  key  constructs,  domain  frameworks,  heuristics,  and  prompts  that  also 
serve  to  guide  the  process.  We  were  also  impressed  by  how  similarly  the  scientists  and 
military  experts  approached  problems  in  their  very  different  domains  of  expertise, 
which  allowed  us  to  construct  a  uniform  framework  for  the  analysis  process. 

We  think  the  application  of  this  work  with  the  highest  potential  impact  would  be 
to  develop  a  computer  tool  that  facilitated  expert  analysis,  based  on  the  theory  of 
epistemic  forms  and  games.  As  we  have  argued  elsewhere  (Collins  &  Ferguson,  1993) 
computer  tools  like  STELLA  and  Model  It  (Jackson,  Stratford,  Krajcik,  &  Soloway,  1994) 
support  analysts  to  construct  models,  using  a  single  epistemic  form.  Our  data  support 
the  fact  that  experts  use  a  variety  of  epistemic  forms  in  constructing  their 
understanding  of  a  situation,  and  that  the  models  they  construct  are  based  on  multiple 
epistemic  forms  embedded  within  each  other.  Therefore  we  think  that  the  next 
generation  of  tools  to  support  model  building  will  need  to  provide  multiple  forms  that 
analysts  can  use  to  construct  their  interpretations. 
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